Papers with recurrent models
A logical-based corpus for cross-lingual evaluation (D19-61)
Copied to clipboard
| Challenge: | a recent study shows that deep learning models can be used to solve textual inference tasks using simple linguistic patterns. |
| Approach: | They propose a set of syntactic tasks focused on contradiction detection that exploit linguistic patterns. |
| Outcome: | The proposed tasks can be implemented in English and Portuguese. |
How Well Can a Long Sequence Model Model Long Sequences? Comparing Architectural Inductive Biases on Long-Context Abilities (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in system engineering and model design have enabled extended context models. |
| Approach: | They propose to scale up models that are purported to support extended contexts . they show that recurrent models still suffer in the same settings as long-context LLMs if attention is given to them . |
| Outcome: | The proposed models can extend to infinite sequence length, but they suffer in the same settings as long-context models with attention. |
Recurrent models and lower bounds for projective syntactic decoding (N19-1)
Copied to clipboard
| Challenge: | a string of recent work has attempted to delve into the formal properties of neural network topology choices. |
| Approach: | They propose to use recurrent models to perform projective maximum spanning tree decoding . they also prove the lower bounds of projective maximal spanning trees . |
| Outcome: | The proposed model can perform better than Eisner's model, proving it impossible to predict a projective MST. |
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a tendency to hallucinate false or misleading information, limiting their reliability. |
| Approach: | They examine how architecture-based inductive biases affect the propensity to hallucinate . they find that the models are more reliable and more reliable than traditional models . |
| Outcome: | The proposed models can be used to train and train large language models that are factual or able to explain themselves through their knowledge. |
Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)
Copied to clipboard
| Challenge: | Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units. |
| Approach: | They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors. |
| Outcome: | The proposed model eliminates the decoder and reduces the search space for splitting points. |
Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)
Copied to clipboard
| Challenge: | LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. |
| Approach: | They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input. |
| Outcome: | The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients. |
On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages (2020.coling-main)
Copied to clipboard
| Challenge: | recurrent models have been effective in NLP tasks but performance on context-free languages (CFLs) is weak. |
| Approach: | They evaluate the performance of recurrent models on Dyck-n languages . they find that they are expressive enough to recognize Dyck words of arbitrary lengths if their depths are bounded. |
| Outcome: | The proposed models generalize well on Dyck-n languages, while performing poorly on longer test strings. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found that Transformers struggle to model several formal languages when compared to recurrent models. |
| Approach: | They conduct an extensive empirical study on Boolean functions to demonstrate that Transformers are relatively more biased towards functions of low sensitivity . they also show that Transformer's generalize near perfectly even in the presence of noisy labels whereas recurrent models overfit and achieve poor generalization accuracy. |
| Outcome: | The results show that Transformers generalize near perfectly even in noisy Boolean functions whereas recurrent models overfit and achieve poor generalization accuracy. |
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models of working memory are limited by self-attention, but we propose a new variant of regular programming that can model regular languages. |
| Approach: | They propose a Transformer variant that constructs working memory along the depth dimension . they find it rediscovers the local windowed attention effect needed for length extrapolation . |
| Outcome: | The proposed model rediscovers the local windowed attention effect needed for length extrapolation. |
HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models that memorize past tokens have “flat” memory architectures that restrict the context window. |
| Approach: | They propose a framework that imitates human memorization behavior by preserving tokens from early input segments, passing memory embeddings along the sequence, and recalling relevant information from history. |
| Outcome: | The proposed framework outperforms existing models in language modeling and question-answering tasks and achieves comparable or superior generation quality to long-context models with 2 57 fewer parameters and 2.5 116 less inference memory. |
Shortformer: Better Language Modeling using Shorter Inputs (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods require computationally expensive relative position embeddings. |
| Approach: | They propose two methods that decrease input length to improve perplexity and perplexability. |
| Outcome: | The proposed methods speed up training by a factor of 1.65 and reduce memory usage. |
Relating Simple Sentence Representations in Deep Neural Networks and the Brain (P19-1)
Copied to clipboard
| Challenge: | Existing deep learning models for natural language processing are not fully studied. |
| Approach: | They investigate whether deep recurrent models learn sentences against those encoded by the brain and whether there is any correspondence between hidden layers of these models and brain regions when processing sentences. |
| Outcome: | The proposed models can be used to synthesize brain data and improve subsequent stimuli decoding accuracy. |
On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LSTMs have not revealed their ability to model syntactic properties. |
| Approach: | They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction. |
| Outcome: | The proposed model generalizes well on counter languages and its learned mechanism correlates with it. |
StateX: Enhancing RNN Recall via Post-training State Expansion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that RNNs with large recurrent states are expensive to train . however, the ability to recall contextual information from long contexts is underperforms them in certain aspects. |
| Approach: | They propose a framework that expands the states of pre-trained RNNs by scaling them up to 1.3B . they use a recurrent architecture that compresses contextual information into a fixedsize state . |
| Outcome: | Experiments on models with up to 1.3B parameters show that StateX expands state sizes without incurring high post-training costs or compromising other capabilities. |
Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work on recurrent models for text embedding is limited to small task-specific models. |
| Approach: | They propose a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. |
| Outcome: | The proposed architectures achieve competitive performance across benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based models. |